Time-Sensitive Bandit Learning and Satisficing Thompson Sampling

The literature on bandit learning and regret analysis has focused on contexts where the goalis to converge on an optimal action in a manner that limits exploration costs. One shortcomingimposed by this orientation is that it does not treat time preference in a coherent manner.Time preference plays an important role when the optimal action is costly to learn relative tonear-optimal actions. This limitation has not only restricted the relevance of theoretical resultsbut has also influenced the design of algorithms. Indeed, popular approaches such as Thompsonsampling and UCB can fare poorly in such situations. In this paper, we consider discountedrather than cumulative regret, where a discount factor encodes time preference. We proposesatisficing Thompson sampling – a variation of Thompson sampling – and establish a strongdiscounted regret bound for this new algorithm.